System and method for facts extraction and domain knowledge repository creation from unstructured and semi-structured documents

ABSTRACT

Provided are methods and systems that extract facts of unstructured documents and build an oracle for various domains. The present invention addresses the problem of efficient finding and extraction of facts about a particular subject domain from semi-structured and unstructured documents, makes inferences of new facts from the extracted facts and the ways of verification of the facts, thus becoming a source of knowledge about the domain to be effectively queried. The methods and systems can also extract temporal information from unstructured and semi-structured documents, and can find and extract dynamically generated documents from Deep or Dynamic Web.

CROSS-REFERENCE TO RELATED APPLICATIONS AND CLAIM OF PRIORITY

This application is a divisional of U.S. Ser. No. 13/546,960, filed onJul. 11, 2012, which is a divisional of U.S. Ser. No. 12/833,910, filedon Jul. 9, 2010, which is a continuation of U.S. Ser. No. 12/237,059,filed on Sep. 24, 2008, which is a divisional of U.S. Ser. No.11/152,689, filed Jun. 13, 2005, each of which claim the benefit of U.S.Ser. No. 60/580,924, filed Jun. 18, 2004. All of which are fullyincorporated herein by reference in their entirety.

BACKGROUND

1. Field of the Invention

This invention relates generally to methods and systems for informationretrieval, processing and storing, and more particularly to methods andsystems of finding, transforming and storage of facts about a particulardomain from unstructured and semi-structured documents written in anatural language.

2. Description of the Related Art

The transformation of information from one form to another was and stillis quite a formidable task. The major problem is that the purpose ofinformation generation in the first place is communication with humanbeings. This assumption allowed and forced the use of loosely structuredor purely unstructured methods of information presentation. A typicalexample would be a newspaper article. Sometimes the information ispresented in a little more structured form like in a company's pressrelease, or in SEC 10-K form. But even in the latter case the majorityof information is presented using plain (e.g. English) language. Withthe information explosion there has been, particularly with theInternet, the need for aggregation and automatic analysis of thevirtually infinite amount of information available to the public becameapparent and urgent. The fundamental problem with this analysis is inthe very fact that the information is originated by human beings to beconsumed by human beings. So, to perform aggregation and automaticanalysis of this information a computer needs to transform/translatesemi-structured or completely unstructured text into a structured form.But to do that one needs to create a machine that can understand naturallanguage—this task is still far beyond the grasp of AI community.Furthermore, to understand something means not only to recognizegrammatical constructs, which is a difficult and expensive task byitself, but to create a semantic and pragmatic model of the subject inquestion.

A number of scientists and businesses tried to solve this problem bycreating a statistically generated ontology of a subject area andgenerating tools to navigate the Internet and other sources ofinformation using this ontology and key words. Some of them went evenfurther and generated the “relevance” index to prioritize pieces ofinformation (e.g. web pages) by their “importance” and “relevance” tothe question (e.g. Google™).

The fundamental problem with this approach is that it still does notperform the task at hand—“analyze and organize the sea of informationpieces into a well managed and easily accessible structure”.

Transformation of information contained in billions and billions ofunstructured and semi-structured documents that are now available inelectronic forms into structured format constitutes one of the mostchallenging tasks in computer science and industry. The Internet createda perception that everything one needs to know is at his/her fingertips.Search engines strengthen this perception. But the reality is that theexisting systems like Google™ Yahoo™ and others have two majordrawbacks: (a) They provide only answers to isolated questions withoutany aggregations; so there is no way to ask a question like “How manyCRM companies hired a chief privacy officer in the last two years?”, and(b) the relevancy/false positive number is between 10% and 20% onaverage for non specific questions like “Who is IT director at WellsFargo bank?” or ‘Which actors were nominated for both an Oscar and aGolden Globe last year?” These questions require the system thatcollects facts and then present them in structured format and stored ina data repository to be queried using SOL-type of a language.

The following metaphor can be applied. Keyword search can be viewed as aprocess of sending scouts to find a number of objects that resemble whatone is looking for. The system that converts unstructured data into astructured repository becomes an oracle that does not look for answersbut just has the information ready.

The Internet has been generated by the efforts of millions of people.This endeavor could not be achieved without a flexible platform andlanguage. HTML provided such a language and with its loose standards hasbeen embraced worldwide. But this flexibility is a mixed blessing. Itallows for unlimited capabilities to organize data on a web page, but atthe same time makes its analysis a formidable task. Though there is notheoretical possibility to create an algorithm to analyze page structureof an arbitrary web page, the fact that the ultimate goal of a page isto be read by a human being makes the problem practically solvable.

The Major challenge of the information retrieval field is that it dealswith unstructured sources. Furthermore, these sources are created forhuman not machine consumption. The documents are organized to matchhuman cognition process that is based on using conventions and habitsimmanent to a multi-sense, multi-oracle perception.

Examples of multi-sense perception include the conventions that dictatethe position of a date in a newspaper (usually on the top line of apage, sometimes on the bottom line, or in a particular frame close tothe top of the page) or continuation of the article in the next columnwith the consideration of a picture or horizontal line dividing the pagereal estate into areas. Examples of multi-oracle perception mechanismsinclude the way companies describe their customers—it can be a pressrelease, it can be a list of use cases, a list of logos, or simply alist of names on a page called “Our customers”.

With the increase of throughput the Internet pages become more and morecomplex in structure. Now they include images, sounds, videos, flashes,complex layout, dynamic client side scripting, etc. This complexitymakes the problem of extraction of units like an article quiteproblematic. The problem is aggravated by the lack of standards and thelevel of creativity of web masters. Some hopes can be placed on theemerging semi-structured data feed standards like RSS, but the web pagesthat mimic the centuries old tradition of presenting news on page forhuman eyes are here to stay.

The problem of extracting main content and discarding all other elementspresent on a web page constitutes a formidable challenge. At the momentthe status quo is that the automatic systems that “scrape” articles fromdifferent web sites for consolidation or analysis use so-calledtemplates. Templates are formal descriptions of a way how a webmaster ofa particular newspaper presents the information on the web. Thetemplates constitute three major challenges. Firstly, one needs tomaintain many thousands of them. Secondly, they have to be updated on aregular basis due to ever changing page structures, new advertisement,and the like. Because newspapers do not notify about these changes, themaintenance of templates require constant checking. And thirdly, it isquite difficult to be accurate in describing the article, especially itsbody, since each article has different attributes, like the number ofembedded pictures, length of title, length of body etc.

Temporal information is critical for determination of relevancy of factsextracted from a document. There are two problems to be addressed. Oneis to extract time stamp(s) and another one is to attribute the timestamp(s) to the extracted facts. The second problem is closely relatedto the recognition of HTML document layout including determination ofindividual frames, articles, lists, digests etc. The time stampextraction process should be supplemented with the verificationprocedure and strong garbage model to minimize false positive results.

A time stamp can be either explicit or implicit. An explicit time stampis typical for press release, newspaper articles and other publications.An implicit time stamp is typical for the information posted oncompanies' websites, when it is assumed that the information is current.For example, executive bios and lists of partners typically have animplicit time stamp. The date of a document with an implicit time stampis defined as a time interval when a particular fact was/is valid.

Implicit time stamp extraction is straightforward. When a fact isextracted from a particular page for the first time, the lower bound ofthe time interval is set to the date of retrieval—we can assume that thefact was valid at least at the day of retrieval and possibly earlier. Atthe same time the upper bound of the time interval is also set to thedate of the retrieval—we can assume that the fact was valid on the dayof retrieval. As the crawler revisits the page and finds it and thefacts unchanged the upper bound of the time interval is increased to thedate of the visit (the fact continues to hold on the date of the visit).

Explicit time stamps are much harder to extract. There are three majorchallenges: (1) multi-document nature of a web page; (2) no uniform ruleof placing time stamps and (3) false clues. Typical examples of amulti-document page are a publication front page in a form of a digestor a digest of a company's press releases.

In the case of newspaper, the convention is that the top of the pagecontains the today's date, and all articles are presumed being timestamped with this date. The situation with a web page is much morecomplex, since with the development of convenient tools for web pagedesign people became quite creative. Nevertheless, the overall purposeof the web page—to distribute information in a way convenient to areader—keeps the layout of a page from becoming completely wild. That iseven more applicable to business-related articles, where the goal is toproduce easily scannable documents for busy business readers. In mostcases, the time stamp of an article is positioned at the top of adocument, while the documents on the page are positioned in a sequentialorder looking from html tags prospective.

The variety of the ways how documents created by humans represent thesame facts, demands the system that needs to recognize and extract themto be a hybrid one. That is why homogeneous mechanisms can not functionproperly in an open world, and thus rely on constant tuning or onfocusing on a well defined domain.

For a long time, the main thrust in Information Retrieval field was inbuilding mechanisms to deal with the ever growing amount of availableinformation. With the explosion of the Internet, the problem ofscalability became critical. For keyword based search systems,scalability is straightforward. For a system of facts, extraction likeBusiness Information Network, the problem of scalability issignificantly more complex. That is because facts about the same objectoccur in different documents, and thus should be collected separatelybut used together to infer additional facts and to verify or refute eachother, and to build a representative description of an object.

The original premise of Information Retrieval was to create mechanismsto retrieve relevant documents with as low as possible number of falsenegative (missed) and false positive (not-relevant) ones. All existingsearch engines are based on that premise with the emphasis on low falsenegative part. The relevancy (false positive rate) of search results isa very delicate subject, which all search vendors try to avoid. As amatter of fact, independent studies showed that a typical keyword searchof a business person like “Wells Fargo”+“IT Director” generates up to athousand url links out of which just 10% are relevant and even they arelocated all over the place; the probability to see a relevant link inthe first page of search results (first 10 links) is practically thesame as the probability to see it on the 90th page (links 900 to 910).As opposed to search engines, the system that provides answers simplycan't afford to have high level of false positive rate. The systembecomes useless (unreliable) if false positive rate is higher than asingle digit. To provide that level of quality, the system should employspecial protective measures to verify the facts stored in itsrepository.

URL-based (static) Internet currently consists of more than 8 billionpages and grows with the speed of 4 million pages per day. These do notreflect so-called Deep Web or dynamically generated request-response webpages that represent one order of magnitude more than the staticInternet. That humongous size of the search space presents significantdifficulty for crawlers, since it requires hundreds of thousandscomputers and hundreds of gigabits per second connections. There is avery short list of companies like Google™, Microsoft™ Yahoo™ and AskJeeves™, which can afford to crawl the entire Internet space (staticpages only). And if the task is to provide a user with a keywords indexto any page on the Internet, that is the price to pay. But for manytasks that is neither necessary nor sufficient.

If one looks at the problem of using the Internet as a source of answersto a particular set of questions and/or to use the Internet to provideinformation to a particular application, the desire is to look only at“relevant” pages and never even visit all others. The problem is how tofind these pages without crawling the entire Internet. One of thesolutions is to use search portals like Google™ to narrow the list ofpotentially relevant pages using keyword search. That approach assumesadvance knowledge of keywords that are used in the relevant pages. Alsoit assumes that third party (Google™ et al.) database can be used to domassive keyword requests. Also the number of pages to be extracted andto be analyzed can significantly supersede the number of relevant pages.

Static Internet constitutes just a small fraction of all documentsavailable on the Web. Deep or dynamic web constitutes a significantchallenge for web crawlers. The connections between web pages arepresented in a dynamically generated manner. To define the question, theDHTML forms are used. The page that is rendered does not exist and isgenerated after the request for it is made. The content is typicallycontained in the server database and the page is usually a mix ofpredefined templates (text, graphic, voice, video etc.) and the resultsof dynamically generated database queries. Airlines web sites provide avery good example of ratio between static pages on web site and theinformation available about flights. Online dictionaries show even moredramatic ratio between the size of surface and deep web, where the deepweb part constitutes 99.99% while the static web part is mere 0.01%.

Since the main issue in dealing with the dynamic web is that the answeris rendered only to the rightfully presented question, a mechanism thatdeals with the Deep Web should be able to recognize what type ofquestions should be asked and how they should be asked, and then be ableto generate all possible questions and analyze all the answers. At themoment Deep Web is not tackled by the search vendors and continues to bea strong challenge.

Typical examples are travel web sites and job boards. Furthermore, nowpractically any company website contains forms, e.g. to present the listof press releases. The major problem is to find out what questions toask to retrieve the information from the databases, and how to obtainall of it.

NLP parsing is a field that was created in the 1960's by N. Chomsky'spioneer work on formal grammars for natural languages. Since that time,a number of researches tried to create efficient mechanisms to parse asentence written in a natural language. There are two problemsassociated with this task. Firstly, no formal grammar of a naturallanguage exists, and there are no indications that it will ever becreated, due to the fundamentally “non-formal” nature of a naturallanguage. Secondly, the sentences quite often either do not allow forfull parsing at all or can be parsed in many different ways. The resultis that none of the known general parsers are acceptable from thepractical stand point. They are extremely slow and produce too many orno results.

Dictionaries play an important role in facts verification. The mainproblem though is how to build them. Usually some form of bootstrappingis used that starts with the building of initial dictionaries. Then aniterative processes use dictionaries to verify new facts and then thesenew facts help to grow dictionaries which in their turn allow extractingmore facts etc. This general approach though can generate a lot of falseresults and specific mechanisms should be built to avoid that.

At the same time, even if the parser quickly generated a grammaticalstructure of a sentence, it does not mean that the sentence contains anyuseful information for a particular application. Semantic and pragmaticlevels of a system are usually responsible for determination ofrelevancy.

One of the most difficult problems in facts extraction in InformationRetrieval is the problem of identification of objects, their attributesand the relationships between objects. A typical information systemcontains a pre-defined set of objects. The examples are abundant. Adictionary is a classic example with objects being words chosen by theeditors of the dictionary. In business information systems likeHoover's, the objects include a pre-defined list of companies. But ifthe system is built automatically, the decision whether a particularsequence of words represent a new object is much more difficult. It isespecially tricky in the systems that analyze large number of newdocuments on a daily basis creating significant restrictions on the timespent on the analysis.

Thus, when a knowledge agent extracts a potential object, relationshipor attribute, the more strict its grammar the less the number of falsepositives it produces. On the other hand, strictness of grammar limitsits applicability. The success of the recursive verification depends onthe level of heterogeneity of knowledge agents and the presence ofdocuments describing the same objects using different grammaticalconstructs. The latter is quite typical for the Internet whileheterogeneity depends on the system design.

An information system built from unstructured sources has to deal withthe problem that objects and facts about them come from disparatedocuments. That makes identification of objects and establishing theequivalency between them a formidable task. Thus, if a web pagecontaining an article describes a company as IBM while another onementions International Business Machine, somehow the facts from botharticles should be attributed to the blue chip company that is traded onNew York Stock Exchange under the ticker IBM, has IRS number 130871985and is headquartered in Armonk, N.Y. To be able to establish suchdetermination special mechanisms should be developed.

A major challenge with facts extraction from a written document comesfrom the descriptive nature of any document. While describing a fact thedocument uses names of objects, not objects themselves. Thus, factsextraction faces a classic problem of instances vs. denotatum. There isno universal solution for that problem available. On the other handsince the purpose of the business-related documents is to communicate amessage, there are rules that writers of these documents follow. Forexample, inside one document two different companies are not called bythe same name (e.g. Aspect Communications and Aspect Lab will not bereferred simply as Aspect if both are described in the same document,while the word Aspect can be used extensively in the document describingjust Aspect Communications). Another important rule based on the factthat the object should be well defined; otherwise the message isconfusing. In the case of a company, there is usually a paragraphdescribing the details about the company, such as the “About” section ina press release, or information about a company's location or its URL.Similar narrowing mechanisms are used for people. For example,mentioning of a person is done in a following way: “ . . . ”, said JohnSmith, vice president of operations at XYZ.com. Again, if the mechanismsare applied to a narrower domain the object identification proceduresare easier to deal with than in a more general case.

Another challenge with such a system is that it should have mechanismsto go back on its decision on some equivalence without destroyingothers. To provide object identification and equivalence the inferencemechanisms should be incorporated into the system.

One of the most common ways to introduce a person in an article isthrough the mentioning of the person's name, work affiliation andhis/her quotes. This is how news articles and press releases are usuallywritten. This “communication standard” constitutes one of the mainsources of Business Information Network-related facts.

Quantitative information plays very a significant role in InformationRetrieval. In the majority of the unstructured documents, thequantitative information in the form of numbers associated with aparticular countable object. These numbers represent important pieces ofinformation that are used to describe the detailed information relatedto the facts described in the document. We call these numbers VINs, VeryImportant Numbers. The examples of VINs in the case of business factsare: number of employees in a company, number of customerrepresentatives, percent of the budget spent on a particular businessactivity, number of call centers, number of different locations, age ofa person, his/her salary etc. If an information system has VINs in it,its usability is significantly higher. VINs always represent the mostvaluable part of any market analysis, lead verification, and salescalls. The countable objects VINS constitute a significant pool ofinformation that helps to make right business decisions.

Extraction of entities and their relationships from a text, news articleor product description, is done by using local grammars and islandparsing approach. The problem with local grammars is that they aredomain dependent and should be built practically from scratch for a newdomain. The challenge is to build mechanisms that can automaticallyenhance the grammar rules without introducing false positive results.

For a long time, information systems vendors built the systems that hadone kind of objects. The examples are people telephone directories,yellow pages etc, where the objects are individuals and businessesrespectively. Practically the same principle is used by businessinformation systems offered by D&B, Hoovers and others. Socialnetworking systems existing on the market today typically apply theconcept of relationship to one type of objects—people. Since business isdone with people and companies together, Business Information Network'sknowledge about the relationships between people, people and companiesand between companies brings the level of adequacy and sophistication toa completely different level. The questions like “which company from myprospect list recently employed a CIO that worked for one of mycustomers over last 3 years” are completely beyond the capabilities ofexisting systems. Two examples of new level of information that can beused if Business Information Network database is built include ImplicitSocial Network and Customer Alumni Network as introduced in thisinvention.

In any market economy, the livelihood of the company depends on itsrelationships with the outside world, its internal infrastructure, itsemployees and vital activity parameters, such as cash flow and profit.Short of reading people's minds and perusing through proprietarydocuments, the Internet provides the best shot at all these factors thatdescribe companies and its place in the economy. Knowing these facts isuseful in many areas, e.g. it empowers sales and business developmentpeople. The mentioned facts can significantly improve their business andincrease effectiveness of the economy at large. As previously discussed,because the companies are interested in promoting themselves, theywillingly publish a lot of information, and the Internet made it easierfor the publishers and for the receivers of this information. Theproblem is how to extract the relevant facts from billions of web pagesthat exist today, and from tens of billions pages that will populate theInternet in the not so distant future.

Thus, there is a clear need for methods and systems, for particulardomains, that extract facts from billions of unstructured documents.There is a further need for methods and systems that address the problemof efficient finding and extraction of facts about a particular subjectdomain from semi-structured and unstructured documents. Yet there isanother need for methods and systems that provide efficient finding andextraction of facts about a particular subject domain and makeinferences of new facts from the extracted facts and the ways ofverification of the facts. There is yet another need for methods andsystems that provide efficient find and extraction of facts about aparticular subject domain that create an oracle that uses structuredfact representation and can become a source of knowledge about thedomain to be effectively queried.

SUMMARY

Accordingly, an object of the present invention is to provide methodsand systems that extract facts from billions of unstructured documentsand build an oracle for various domains.

Another object of the present invention is to provide methods andsystems that address the problem of efficient finding and extraction offacts about a particular subject domain from semi-structured andunstructured documents.

A further object of the present invention is to provide methods andsystems that can efficiently find and extract facts about a particularsubject domain and make inferences of new facts from the extracted factsand the ways of verification of the facts.

Still another object of the present invention is to provide methods andsystems that can efficiently find and extract facts about a particularsubject domain, which create an oracle that uses structured factrepresentation and can become a source of knowledge about the domain tobe effectively queried.

Still another object of the present invention is to provide methods andsystems, which can extract temporal information from unstructured andsemi-structured documents.

Still another object of the present invention is to provide methods andsystems, which can find and extract dynamically generated documents fromso called Deep or Dynamic Web that contains today tens of billion ofdocuments.

DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram of an embodiment of a system.

FIG. 2 shows the overall system architecture.

FIG. 3 describes the process for finding relevant unstructured andsemi-structured documents, extraction of facts from them, verifying themand storing them in the repository.

FIG. 4 describes the process of effective crawling of the web using theconcept of crystallization points.

FIG. 5 describes the method of automatic DHTML form detection andcrawling of Deep (Dynamic) Web.

FIG. 6 provides a detailed description of false negative rate reductionin crawling by automatic determination of CP crawling parameters.

FIG. 7 provides a detailed description of the process of extracting apage layout from HTML pages.

FIG. 8 describes the process of determining of time reference for facts.

FIG. 9 describes the process of sentence parsing based on the conceptsof island grammar.

FIG. 10 provides the description of the multi-pass bootstrapping processto increase precision of the fact extraction.

FIG. 11 describes the process of extractionperson-position-company-quote facts from unstructured text.

FIG. 12 describes the process for detection and extraction of VeryImportant Numbers and corresponding objects.

FIG. 13 describes the process of automatic expansion of grammar rulesusing iterative training.

FIG. 14 describes the three-layer system of object identification.

FIG. 15 describes the process of recovery from object identificationerrors.

FIG. 16 illustrates the types of relationships in Business InformationNetwork.

FIG. 17 illustrates the process of generation of Business InformationNetwork.

FIG. 18 illustrates the concept of Implicit Social Network.

FIG. 19 illustrates the concept of Customer Alumni Network.

DETAILED DESCRIPTION

The present invention includes a method and apparatus to find, analyzeand convert unstructured and semi-structured information into astructured format to be used as a knowledge repository for differentsearch applications.

FIG. 1 is a high-level block diagram of a system for facts extractionand domain knowledge repository creation from unstructured andsemi-structured documents. System 10 includes a set of documentacquisition servers (12, 14, 16 and 18) that collect information fromthe World Wide Web and other sources and using surface and deep webcrawling capabilities, and also receive information through direct feedsusing for example RSS and ODBC protocols. System 10 also includes adocument repository database 20 that stores all collected documents.System 10 also includes a set of knowledge agent servers (32, 34, 36 and38) that process the document stored in the database 20 and extractcandidate facts from these documents. The candidate facts are stored inthe candidate database 40. System 10 also includes inference andverification servers (52 and 54) that integrate and verify candidatefacts from the database 40 and store the results in the knowledgedatabase 60. The database 60 can be used as a source for data feeds andalso can be copied to a database server for an internet application,such as a business information search, job search or travel search.

In one embodiment, the search application is a Business RelationshipNetwork that is a system that finds, analyzes and converts unstructuredand semi-structured business information present in the World Wide Web,and provides new generation search capabilities for the internet users.

For a long time, the main thrust in the Information Retrieval field wasin building mechanisms to deal with the ever growing amount of availableinformation. With the explosion of the Internet, the problem ofscalability became critical. For keyword base search systems,scalability is straightforward. For a system of facts extraction, likeBusiness Information Network, the problem of scalability issignificantly more complex. That is, because facts about the same objectoccur in different documents, and thus should be collected separatelybut used together to verify or refute each other, and to build arepresentative description of an object.

In one embodiment of the present invention as illustrated in FIG. 2, amulti-parallel architecture and algorithms are presented for building alinearly scalable system for Information Retrieval that can not onlyindex documents but can extract from them facts about millions ofobjects.

The architecture of the system 10 is based on the principles ofindependency of different levels in the system and independency withinlayers. Thus crawling is done independently from the analysis of thepages. Knowledge agents work independently from each other and withinthe context of an individual page. Only after candidate facts areextracted they are compared against each other during the inference andverification phase. At that time, the size of the task is several ordersof magnitude lower than originally, so it can be handled with limitedresources. The algorithms are closely related with these concepts ofindependent knowledge agents and deferred decisions described hereafter.These principles that are implemented in building Business InformationNetwork are applicable to many other areas, such as job listings, travelinformation, and legal information.

In one embodiment of the present invention, methods and systems areprovided, as illustrated in FIG. 3, that process facts extraction anddomain knowledge repository creation. In one embodiment, the methods andsystems of the present invention utilize the following steps. Firstly,crawlers crawl the Internet and other sources and generate a set ofdocuments that are analyzed by knowledge agents. Then each document isanalyzed by one or more knowledge agents. The analysis consists of twoparts—global analysis/layout recognition and local analysis. The resultsof the analysis are facts that are scrutinized by further steps toeliminate false positives. Then each fact goes through the inferencestage where it is getting associated with other facts and existingobjects in the repository. After association, the facts are scrutinizedagainst each other to eliminate duplicates and false positives, andfinally the facts that passed through previous steps are stored in therepository that becomes a domain oracle.

In one embodiment of the present invention, a method is presented forreduction of the number of false positives in the fact extractionprocess in Information Retrieval. The mechanisms are based on theprinciples of deferred decisions and iterative verification. By way ofillustration, and without limitation, this method is illustrated usingBusiness Information Network examples, but has general applicability.

The problem of false positives is much more severe for facts-basedinformation system as opposed to search engines. To decrease andeventually eliminate the number of false positives, the decision makingprocess should have several safety mechanisms. The more heterogeneousthese mechanisms are, the more reliable the overall system is. Thedetails of building hybrid systems in Information Retrieval aredescribed hereafter. When a hybrid or a multi-oracle system makes adecision, it is more reliable than the decision of a pure homogeneoussingle-oracle system. But there is another dimension that increases thereliability of a decision—to defer it until new information isavailable. The deferred decision was used quite successfully, forexample, in speech recognition systems. The acoustic cues and theresults of phoneme recognition are later used at the linguistic level.The same mechanisms can be applied to the fact extraction in InformationRetrieval.

By way of illustration, Business Information Network PPCQ knowledgeagent, see below, produces candidate parses while at the database leveldifferent parses are checked versus each other and versus establishedfacts in the Business Information Network database to find out whichcandidates represent a new fact, and which ones indicate a potentialcontradiction with the existing facts, and therefore should bescrutinized by verification process.

The discrepancy between different candidates for facts and inconsistencybetween the new and existing facts constitute the area where deferreddecisions principle shows its ultimate power.

When these situations occur, the presence of all evidences, parametersextracted by knowledge agents at all stages of the fact extractionprocess, allows for cross references and elimination of the incorrectcandidates. If the existing evidences are not sufficient to resolve thediscrepancy or eliminate a candidate with certainty, the followingiterative process can be applied to extract additional parameters.Typically when knowledge agents produce a candidate they supply the nextlayers with just the necessary parameters such as confidence level. Inmany cases, the output is the best result as opposed to N-best results.Next, layers do not have knowledge or even understanding of specificsand have to rely on this limited number (usually 1) factor. And usuallythe decision ends up being done based upon this insufficientinformation. If there is a way to ask the knowledge agent again and, forexample, ask for several best results and then combine the originalfactors that constituted the final score with the factors generated bynext layers, the decision becomes much more reliable. Thus deferring thedecision, submitting N-best instead of the best answer, and thecapability to return back and check the reasons for the choice of thebest answer creates a system with low false positives.

Business Information Network utilizes these principles in many cases.PPCQ does not make a decision in the case of embedded parses, but rathersubmits all of them to the next layers. These layers provide databaseand dictionary verifications and choose the best candidate. Anotherexample would be for the time stamp Knowledge Agent when thecontradiction in bio can require considering all candidates for timestamp in the document, and choose the one that eliminates thecontradiction, or if the time stamp ends up being correct to inferpotentially false positive fact in the database.

False negatives and false positives are typically perceived as being apart of a zero sum game. You can decrease one, but at the same time theother one will increase. The main reason of that is in the fact that theused mechanisms are homogeneous and non-iterative. In one embodiment ofthe present invention, a method is presented for a solution of thatproblem in the Information Retrieval space.

To get out of the predicament of a zero-sum game two principles areutilized: use of heterogeneous Knowledge Agents and Iterative Analysis.

In one embodiment of the present invention, a method is presented forbuilding hybrid systems in Information Retrieval, and their applicationto a particular field of information retrieval of business information.It also addresses the problems of multi-sense multi-oracle perception bydefining two types of mechanisms, statistical and rule-based, ofintegration of results and mutual influence in the decision makingprocess of different types of oracles/KAs and illustrates theseprinciples on the example of hybrid layout recognition system.

The interrelations between different oracles/knowledge agents inInformation Retrieval depend on their nature and their reliability whenapplied to a particular type of a document. In a case of homogeneousKnowledge Agents, e.g. Link-based and Fact-based ranking, a weighted sumof their results produces much more accurate results, while in a case ofheterogeneous Knowledge Agents, e.g. Global and Local Grammar, rulebased approach is more productive.

This method of the present invention can include the following: methodsfor building a hybrid system in Information Retrieval; hybrid relevancyranking based on integration of the results of independentweight/ranking functions; recursive Knowledge Agents application e.g.Global/Layout Knowledge Agents and Local/Statistical/GrammaticalKnowledge Agents.

In one embodiment of the present invention, a method is presented forbuilding a hybrid system that produces a much higher level ofreliability with a low false positive rate. The mechanisms are based onthe principles similar to ones that are used by humans. They include theincorporation of oracles of different origins (such as global and localgrammars), iterative verification process, special garbage model, anddeferred decisions. The methods are illustrated on Business InformationNetwork system.

There are two major cases of integration of different oracles: ahomogeneous one and a heterogeneous one. The first case is typical for arecognition system with independent ranking mechanisms of hypothesis.Thus in speech recognition several lists of candidate words can bemerged together with a linear combination of weights. Known casesdemonstrate a 30-50% reduction in error rate using this mechanism. Thesame approach is applicable to the fact relevancy function and to thedocument reliability.

Heterogeneous case is quite more complex. The approach used in oneembodiment of the present invention is to first specify the “area ofexpertise” of each oracle and incorporate fuzzy logic (high, medium andlow confidence) in decision making. Thus, if an oracle with the right“expertise” has high confidence and all other oracles with the samelevel of expertise have at least medium confidence, the decision isfinal. If there is a contradiction between oracles of the highestexpertise the fact is escalated to other layers of decision makingincluding potential human interaction.

Layout recognition by humans is an iterative process, where content isused to support visual cues like pictures, horizontal and vertical linesetc. The best results are achieved when both content and layout oracleswork in concert with each other to eliminate false page segmentations.This method of the present invention uses this principle to a largeextent to approach and demonstrates it on extraction of such importantcues like ‘about’ clause, address, phone number, time stamp, customersand others from HTML pages.

A set of knowledge agents is created that can provide extremely lowfalse positive rate, and is complimentary to each other. Beingcomplimentary, means that the documents that can not be analyzed by oneof the knowledge agents can be analyzed by others. The trick is how toproduce a set of low false positive knowledge agents that will cover themajority of “relevant” documents. Since each knowledge agent ishomogeneous, the process is similar to covering a square with a set ofdifferent circles. Since all knowledge agents have a low false positiverate, the overall system has both low positive and low negative rates.

These two principles can be widely implemented in Business InformationNetwork. Thus, knowledge agents are built using combination of differentmethods, e.g., page layout recognition algorithms use image processingapproach, while local grammars are built on the principles of NaturalLanguage Processing, while relevancy oracles are statistically based.Recursive verification is used widely across the board, such as factextraction done by knowledge agents influence crystallization pointsbeing used for crawling.

In one embodiment of the present invention, as illustrated in FIG. 4, amethod is provided for efficient crawling of the Internet to find pagesrelevant to a particular application. The examples of the applicationsthat can strongly benefit from these methods include but are not limitedto, business, legal, financial, HR information systems, and the like.The methods can be demonstrated on Business Information Network—BusinessIntelligence information system. In one embodiment, a set of initialURL's (“crystallization points” or CPs) and the recursive rules ofcrawling from them are defined as well as the rules of adding newcrystallization points to crawl from. Any mechanism of partial crawlingcan potentially miss relevant pages. The right combination of theparameters for four major steps defined below can be achieved by commonsense supported by experiments. But even if the initial set of CPs isrelatively small, and the crawling rules are relatively stringent, thereis always a way to expand both and also the CP extension provides forthat. The only restricting factor is the capacity of the datacenter andthe available bandwidth. To decrease the false positive rate specialiterative mechanisms are introduced.

For a particular application, such as Business Information Networksystem, service the Internet can be divided into the following parts:companies/organizations web sites; business publications like magazines,conference proceedings, business newspapers; general purposenewspapers/information agencies; others including personal web sites,blogs, etc.

The first two parts have two advantages, (i) most of the web pagesbelonging to these sites are relevant to Business Information Networkand (ii) they constitute a relatively small percentage of the Internet.

The third source can be extremely relevant or can be completely useless.Fortunately, the sheer volume of the information is significantlysmaller than the Internet. That allows for using two approaches, (i) theuse of keyword search such as the Wall Street Journal archive, or (ii)the use the same approach as with the companies' websites (describedbelow). The fourth source constitutes the majority of the Internet andat the same time is less reliable and is less relevant.

Since the introduction of DHTML standard, crawling mechanisms have todeal both with surface web (static) pages and with the deep web (dynamicpages). At the moment the dynamic web is assessed as containing 90% ofthe information available online.

In one embodiment of the present invention, a method is presented forusing crystallization points to build an effective and efficient WebCrawler. FIG. 4 illustrates one embodiment of a method of crawling usingcrystallization points.

Initial CP's depend on the application, but usually are easy to obtain.For Business Information Network the list consists of the Urls ofFortune 10,000 companies' web sites and 1000 business publications'websites.

A relevant page can be added to the list of CPs if it has the followingfeatures, (i) more than four relevant links, or less than four but to orfrom an “important” page, (ii) it contains a link to a CP, and (iii) therelevance is determined by an independent mechanism, e.g. KnowledgeAgents.

A link (Href in html) is called relevant if it or its descriptioncontains keywords from a predefined list. In case of BusinessInformation Network, this list can include keywords such as “customer”,“vendor”, “partner”, “press release”, “executive”, and the like.

Because relevant information is not necessarily defined on the mainpage, but rather deeper in the site, it is necessary to explorenon-relevant links. At the same time the relevant pages are in mostcases no deeper than 2-3 levels down from the main page. Thus, there aretwo major parameters for pruning, (i) forced depth—the maximum distancefrom a CP without checking relevancy, and (ii) maximum depth—the maximumallowed distance from a CP.

The crawl starts with the initial set of CPs. In one embodiment, thecrawl is done breadth first, meaning that all links from a particularpage are first explored then each one of them is used as a startingpoint for the next step. A URL is considered a terminal node of crawlingif it does not have “relevant” links and the distance from it to the CPsis equal to the predefined “forced depth” (typically 2 or 3, no morethan 4). If a web site has a site map page, which typically has a linkfrom the main page, the forced depth can be just 1.

The crawl stops if one of the following is true, (i) a page is terminal,or (ii) the maximum distance from CPs is reached.

In one embodiment of the present invention as illustrated in FIG. 5, amethod is presented for building a deep web crawler. In one embodiment,the process of deep web crawling is separated in four distinct steps,(i) scout, (ii) analyzer, (iii) harvester, and (iv) extractor.

The scout randomly “pings” the forms to collect dynamic pages behindthem. The analyzer, with the use of the extractor, determines theunderlying structure of queries and generates the instructions for theharvester. The harvester then systematically puts requests to the serverand collects all available pages from the server. The extractor extractsunstructured and semi-structured information from the collected pagesand converts them into a structured form.

The scout crawling rules are divided into dealing with static anddynamic pages. Since any dynamic web site has static pages also, bothtypes of pages should be crawled over by the scout. The static pages arecrawled based on the principles discussed in a description of a genericcrystallization point based crawler elsewhere in this patent. Aspreviously mentioned, the main problem with the dynamic pages is thatthey exist virtually, i.e. they are generated by the server after thequestion is asked. Dynamic HTML standard provides a special mechanism toask a question. The mechanism is called forms. Forms are specialelements of DHTML that have several types of controls allowing fordifferent ways to ask the question. There are options-based controls(e.g. select and inputradio), where a person chooses one of the optionsfor controls, and there are free form controls (e.g. inputtext andtextarea), where any sequence of symbols can be entered. A form cancontain any number of controls.

To know what question to ask, the following statistical approach can beused. A number of questions are chosen that covers all possible patternsof dynamic pages produced by this form to allow the following steps, theanalyzer and the harvester, to create exhaustive enumerations ofquestions that will generate all dynamic pages that the server canproduce. One needs to realize that some questions can produce a subsetof answers of other questions and the answers to different questionsoften overlap. For example, in many cases the default option means “showall”, and using it alone produces all dynamic pages behind the form. Inother cases the options provide alternative answers, like if one choosesstate in job search. In many applications (e.g. travel search) onlyoption-based controls are used.

To deal with unrestricted text-based controls the following set ofquestions represents a good strategy: “*”, “a*”, “b*”, “c*” . . . “z*”.Randomly chosen, these questions most likely generate a representativeset of answers for analyzer and harvester to recon with. Also, amanually created list of questions can be used. This approach worksespecially well for the applications that have a reasonable number ofdynamic pages (within thousands) or have a large number of homogenousdynamic pages like airline ticket search of job boards' sites.

The following table shows an example of the set of rules that can bespecified for the scout. The scout applies these rules to a valid formthat a current crawled page contains. A separate set of rules definewhat forms are considered valid, and is described below.

Run Control Pos Neg Number Number Type KWs KWs Input of Trials 1 SelectJob/openings 5 1 InputRadio Location 4 2 InputText DescriptionA*\ab*\c\d* 3 3 InputRadio Month 4

The rules for choosing random questions are defined by the table likethis. All controls having the same Run Number are mapped to the validcontrols in a valid form. The control is valid if its descriptioncontains one of the positive keywords and does not contain any of thenegative keywords. The map of the rules in the same run to the validcontrols generates a bipartite graph. The scout enumerates all possibleone-to-one pairs of the rules and controls in the graph. For each map itthen generates random choices of options, inputs for text control. Thusfor the Run 1 it is 5*4=20 random choices from Select and InputRadiocontrols, while Run 2 will generate 3 random entries from the list inInput column of the table. This procedure is applied independently toall valid forms on the current HTML page. All HTML pages generated bythese questions are stored for the future scrutiny by the analyzer.

The analyzer takes a set of pages created by the scout and builds theset of rules for the harvester. All pages generated by the scout arepushed through the extractor that extracts facts from these pages andstores them in a database. The set of pages extracted by the scoutrepresent a navigation graph that is also stored in the database. Thus,the analyzer starts with the scouting Navigation Graph (SNG) of pagesand the set of relevant (to the application at hand) facts extractedfrom these pages. This graph constitutes a sub-graph of all relevantpages and paths to them that virtually exist. The problem is to convertthis graph into a set of navigation rules for the harvester to collectall the relevant pages and build full Navigation Graph of dynamic pages.

The Harvesting Navigation Rule Graph (HNRG) is presented as a set ofpaths from the roots, which can be main pages of particular sections ofcompanies' web sites, to the relevant pages (e.g. individual jobpostings). The following procedure is used to build the HNRG from theSNG.

Two relevant pages/nodes in the SNG are called equivalent if they belongto the paths of the same length that contain the same forms and coincideup to the last form. The equivalence class of the relevant nodesconstitutes one rule in the HNRG. The rule is described as a path fromthe root to the form and the number of steps after the last form to getto the relevant nodes. The rule also specifies invalid hyperlinks toavoid excessive crawling without any purpose.

The harvester takes the HNRG and follows one rule at a time. When ithits the form node it applies each combination of options/inputsdetermined by the HNRG and then proceeds with the static crawlingobeying the rules for negative hyperlinks (URLs) and the forced depth ofcrawl. The results are stored similarly to the results of the scout tobe used by the extractor to extract facts.

Any system that can convert unstructured and semi-structured pages canbe used as an extractor. For the analyzer stage sometimes even a binaryoracle that determines “adequacy” of the page is sufficient, but in manycases the oracle of that kind is almost as difficult to build as a realextractor. The extractor that is used in this embodiment is a hybridsystem that uses elements and algorithms described in other parts ofthis patent. Thus, for a job search application the same steps were usedas for the Business Information Network application. Namely, the layoutof a page is extracted. That produces the elements containing job title,job description and job location. Then, the time stamp is extracted.Then, the local grammar is applied to determine the title of a joboffering, and the detailed structure of job location. This informationin combination with the company location being extracted (see BusinessInformation Network) is stored in a Job Database to be used by theend-users to search or by a third party to incorporate into theirconsumer web site. The same database is used by the analyzer to buildthe navigation graph for the harvester, but of course the analyzer dealswith much smaller set of pages that were produced by the scout.

In one embodiment of the present invention, a method is presented forthe reduction of the number of false negatives without going to theother extreme and crawl the entire web. Firstly, the crawling depth andparameters are tuned using training procedures on small samples of theInternet. Secondly, the list of keywords that determines the hyperlinks'relevancy is trained in a similar manner. And thirdly, other statisticalmethods of determining relevancy such as the number of companiesmentioned on the page are applied.

Furthermore, the very structure of the Web with a large number ofhyperlinks between individual pages is quite useful to reduce falsenegatives in crawling. Thus, if a relevant page is too far from certaincrystallization points, and is missed in the initial pass of crawling itis quite likely that it is close to the further rounds of CP extensions.

The parameters for CP crawling can be defined manually for some tasks,but for others it is not feasible due to lack of standards anduniformity in the ways how web pages are linked.

A good example of a quite straightforward determination of crawlingparameters is a case when one needs to crawl a company's website, (andstay within it), and there is a site map page, the page that containslinks to all static pages on the site. Then the depth of crawling of thesite is equal to 2, since the site map page is typically connected tothe home page, and the crawling of static part of the site is reduced tomaking one step to the site map page and then to all other pages in onestep. If the site does not have a site map or if the crawl is notrestricted to one domain at a time, which is typical for BusinessInformation Network, then other means of making CP crawling efficientshould be developed.

In one embodiment of the present invention as illustrated in FIG. 6, analgorithm is provided that generates the CP crawling parameters using arandom walking from a CP.

The algorithm consists of the following steps. The crawl is organized asa breadth-first search with the depth and valences of urls beingbalanced such that the overall size of the search graph is limited by apre-defined number, typically, 1000. Application specific ontologydefines a list of “positive” and “negative” keywords. For example forjob posting application the words “career”, “job”, and “employment”would be in the list of “positive” keywords.

The links are divided in two categories—a) ones that contain “positivekeywords” and do not contain “negative keywords” in the url itself or inthe description of the url, and b) other links that are chosen randomly.The links from the first group are used as soon as the size of the crawlgraph is within the limit defined above independently on the distancefrom the CP. The random links are used only if the distance from the CPdoes not exceed a predefined number, which can be 4 or 5. Using asemi-random walk a directed graph G of pages is generated. Then thepages from the graph G are submitted to the analyzer that determinestheir relevancy to the application at hand, see the analyzer in Deep WebCrawling.

The pages that contain relevant information and the path from the CP tothem represent a subgraph H of the graph G. Then the histogram of thewords that were used in the edges of the graph H is built. The words,excluding auxiliary words like prepositions, and that were used morethan in predefined percentage of the cases, which can be 20%, are addedto the list of “positive” keywords. The words or sequences of words,excluding auxiliary words like prepositions, that were used in the edgesof the graph G\H more than in predefined percentage of the cases, whichcan be 70%, and are used in edges from the graph H in less than apredefined percentage of the cases, which can be 10%, are added to thelist of “negative” keywords. The reason for a much higher threshold isthat “negative” keywords can “kill” the right link and should be managedwith caution.

The maximum depth of the crawl is defined as the maximum of minimaldistances between relevant pages and the root of the graph H—the CP. Theforced depth is defined as the maximum number of links of the secondtype that belong to the shortest paths from the root to the relevantnodes. Since the forced depth parameter controls the percentage ofpotentially irrelevant pages that can be crawled the followingprotective measure is used. If the forced depth parameter exceeds apredefined number, which can be 5, than the histogram of the maximumnumber of links of the second type that belong to the shortest pathsfrom the root to the relevant nodes is built. Then the forced depth isdiminished to the number that covers no less than a predefinedpercentage of links, which can be 80%. Due to the interconnection ofpages on the Internet and the presence of other CPs, this percentage canbe decreased further to 60% if the forced depth is still bigger than 5.The nodes, pages, from the graph H that do not obey maximum depth andforced depth parameters are excluded. The next steps are similar to thebuilding of the Harvesting Navigation Rule Graph defined above.

In one embodiment of the present invention as illustrated in FIG. 7, amethod is presented for automatic high precision/high recall newspaperarticle (Author, Title, and Body) extraction that does not use templatesat all. The articles are assumed to be presented as HTML pages.

The algorithm consists of the following steps. Firstly, an HTML Tree,that includes table depth determination for each node, is built. Thenthe paragraphs are built and the ones contained href, url reference, aredetermined. HTML tags and sheer content of a paragraph are used to markparagraphs that are candidates for authors, titles and dates. E.g. h-tagand title-tag are often used to define a title, b, i, and u-tags areoften used to indicate author, while a paragraph containing a time stampand not much else is a good candidate for the article date, andparagraph consisting of a phrase “written by” and two to five wordsstarting with capital letters is a good candidate for author.

To find the body of an article, the following multi-step procedure isused. Contiguous href and non-href paragraphs are grouped into blocksand are put in three categories by size, small, medium, and large. Smallblocks that are not candidates for Author, Title or Date are excluded.Large blocks, which are separated by one href block with less thanMAXJUMP paragraphs in it, are merged together. Large blocks of the sametable depth, which are separated by no more than MAXJUMP paragraphs.Medium and small blocks with the same table depth, which are separatedfrom the large blocks by no more than MAXJUMP paragraphs, are added tothese large blocks. If a large block does not contains less thanMINLONGLINE number of long lines it is renamed to medium.

Each remaining large block constitutes a candidate for article body.They then are ordered in descending order by their size. If the numberof candidates is 0, the largest medium block that is significantlylarger than the second best medium block, is declared a candidate forthe body of an article. Body candidates that are adjacent to one anotherare glued together. The largest body candidate is chosen as articlebody.

To find a title of an article the following multi-step procedure isused. To recover from the cases of massive attribution of paragraphs astitle candidates, if the majority of the paragraphs within body aremarked as title candidates of the same kind the title flag of that kindis removed from all of them. Then title flags from paragraphs that arebelow the initial large block in the body is eliminated. Title flagsfrom paragraphs with “heavy top”—that have at least MAXABOVETITLEPERC ofbody length above them—are eliminated. If there is a paragraph withtitle flags that is no further than MAXDEPTH2TITLE from the beginning ofthe body, then title flags from paragraphs that are more thanMINDISTTITLES below it are eliminated. If such a paragraph does notexist, title flags from paragraphs inside the body are eliminated. Ifthere are still candidates for a title inside the body the one with theIRScore, if it is larger than MINIRSCORE4TITLE, is chosen at articletitle. IRScore is calculated as the Information Retrieval distancebetween paragraphs and the body.

If there are no candidates inside the body, chose the one with thelargest IRScore as the title. If there are still no valid candidates fortitle, the first paragraph that has IRScore more than MINIRSCORE4TITLEand does not have paragraphs above it longer than MAXCHARINSOFTTITLE, ischosen as article title.

To finalize the results of body, title and author extraction thefollowing multi-step procedure is used. Standard disclaimers like“copyright” paragraphs that contain one of the “prohibited” phrases areeliminated from the body. If the title is extracted, all paragraphsabove it from the body are eliminated. The geometrical boundaries of thearticle are determined to exclude extraneous elements from the articlethat are positioned close to it on the page, or somewhat intersect withthe article. This is done by building a histogram of left and rightcoordinates of each paragraph in the body and choosing two largest picksin it. The information about the position of an HTML element on a screenis determined by rendering it or by relative calculations based uponwidth attribute associated with tables in HTML. The paragraphs with thestart later than the first ⅓ of the body boundaries or end sooner thanthe last 10% of the body boundaries are marked as being non-title. Thesimilar procedure is applied to author candidates. That helpssignificantly to clean up the title and author of the article thusincreasing the overall precision of the layout recognition.

The following values were used in one embodiment of this invention:MAXJUMP=12, MINLONGLINE=3, LONGLINE=50, MINIRSCORE4TITLE=3,MINDISTTITLES=5, MAXDEPTH2TITLE=5, MAXCHARINSOFTTITLE=100,MINTITLELENPERC=0.7, MAXABOVETITLEPERC=0.3.

In one embodiment of the present invention as illustrated in FIG. 8, amethod is presented to solve the problem of time stamp extraction andverification. This method of the present invention presents algorithmsto efficiently detect a potential time stamp, extract it and using thelayout recognition results and immediate extended context of a timestamp, and also the presence of other potential time stamp to determinewhether a particular document has a time stamp and, if it does, toextract it.

Each html page is parsed and represented as a sequence of paragraphs,each associated with its html tag. There are two algorithms implemented.One deals with the multi-document situation, while the other assumesthat there is only one document on a page. Both algorithms use the samemechanism to extract a time stamp from a paragraph. The single documentalgorithm stops when it extracts a valid time stamp and considers itsscope being the entire page. The multi-document algorithm considers eachvalid time stamp having its scope over the paragraph it was extractedfrom and the following paragraphs until the next valid time stamp isextracted. Also these two algorithms differentiate in their garbagemodel. A multi-document algorithm per se does not have the concept ofunknown time stamp for the page. Since the paragraphs are looked at insequential order, if the time stamp is not yet extracted the paragraphin question is declared being with unknown time stamp.

The single document algorithm's garbage model is as follows. As soon asa time stamp is extracted successfully from the current paragraph, theprocess of time stamp extraction for the current page stops, andextracted time stamp is declared as having the scope over the entirepage. That means, for example, that all facts extracted from this pageare assigned with the extracted time stamp. If the page time stamp isnot yet extracted and the current paragraph is “large”, say it has morethan 500 characters, the page is declared as being without a time stamp.The second case of declaring a page being without a time stamp is ifthere is confusion in time stamp extraction in current paragraph.

To extract a time stamp from a paragraph the following multi-stepprocedure is used. Each word, not including separators, is looked at asa potential candidate for Year, Month, or Day of a time stamp. Thecandidate is called strong if it is a candidate for only one out of thethree parts of a date (Y, M, D). Then for each candidate word for Monththe surrounded candidates for Year and Day are checked on whether theyconstitute a triad. Triad is a set of three sequential words inparagraph. The following four (out of potential six) triads areallowed—(Y, M, D), (Y, D, M), (M, D, Y) and (0, M, Y). Quite often thecurrent date is posted on a web page for users' convenience. It can beconfused for the time stamp of a document published on this page. Toavoid that, the triad that is equal to the current date or a day beforeis discarded. For each triad the check is performed on the consistencyof the separators dividing the words in the triad as well as the wordssurrounding the triad being consistent with the time stamprepresentation. The following separators between the words in a triadare allowed: ‘/’ ‘/’, ‘−’, ‘−’, ‘.’ ‘.’, ‘.’ ‘ ’, ‘.’ ‘,’, ‘.’ ‘,’, ‘ ’‘.’, ‘ ’ ‘ ’, ‘ ’ ‘,’, ‘,’ ‘.’, ‘,’ ‘,’, ‘.’‘'’, ‘ ’ ‘'’, ‘,’ ‘'’.

If there is more than one valid triad in a paragraph and they do notshare the same words or words immediately to the left or to the right ofa valid triad are numbers or potential candidates for Year, Month orDay, then the time stamp is declared as unknown.

In one embodiment of the present invention as illustrated in FIG. 9, amethod is presented for efficient grammatical parsing based upon islandgrammar and linear parsing approaches. The results of parsing arerepresented as a sequence of intervals of words in a sentence (notnecessarily including all words in the parsed sentence) marked by thetags defined in the grammar. These tags are later used to determinerelevance of the sentence to the application and potentialintra-sentence references. E.g. anaphora/cataphora resolution and theirspecial case of pronoun resolution, such as in the case when an objectsuch as company or person is named not directly but by a pronoun (he,she, it). In the latter case the noun phrase analyzer is used todetermine the matching between the pronoun and the tagged word interval.

The procedure of grammatical analysis of each paragraph is defined bythe following steps. Firstly, context grammar is applied. Contextgrammar determines the scope of each context on a page. Then aparticular local grammar rule is applied only to the paragraphs thatbelong to the scope of context rules that are related to this localgrammar rule. If the paragraph belongs to the scope of a context grammarrule then all the Local Grammar rules are applied to it. The results ofthe parsing using these rules are considered mapping candidates. Eachcandidate then is checked by applying verification functions. Thesurvived mappings are stored as candidate facts for future analysis byhigher level of the system 10.

The applicability of local grammar rules is determined by a separatelayer—so called Context Grammar The current embodiment of contextgrammar is built as a set of rules each of which has the followingstructure: (LastHeaderHTMLTag, LastHeaderKWs, PositivePrevHeaderHTMLTag, Po sitivePrevHeaderKWs,NegativePrevHeaderHTMLTag, NegativePrevHeaderKWs, Local Grammar RuleType). In some cases, local grammar does not need to be applied, whichis the case, for example, if a table is analyzed. Examples of such rulesare as follows:

(h1\h2\h3\h4\h5\h6\h7\h8\h9\head\strong\b\form\,description\requirement\responsibiliti\qualifications\education\functions\jobsummary\, , , , ,), where local grammar is not applied; or (title\h1\, ,, , KA_LocCity) which defines all paragraphs that are within <title> or<h1> tags scope should be parsed with the local grammar rules of typeKA_LocCity.

Island grammar is described using a special language that allowsspecifying the structure of the sentence in terms of intervals andseparators. The current embodiment of local grammar is built as a set ofrules, each of which has the following structure:

(Separator0, Object1_Type, Object1_Role, Separator1, Object2_Type,Object2_Role, Separator3 . . . ). An example of such a rule is asfollows: (“said”, PersonName, Employee, “,”, PositionName, “of’,CompanyName, Employer, “.”).

A separator can be any sequence of symbols, while roles can be specific(like “employee”, “vendor” etc.) or irrelevant (called “junk”). Anotherexample is related to the context grammar rule described in the previousdiscussion: (city\town\, EMPTY, empty, \, LOCCITY, loccity).

For every grammar rule the following procedure takes place. UsingKnuth-Morris-Pratt algorithm of string matching, the set of all matchesof all words used in the rule to the sentence to be parsed, iscalculated. After that the table of right most possible match of eachword in the rule to the sentence to parse is built. Using this table thelist of all possible parses is built using backward mapping. Thisalgorithm has a complexity of O(nm), where n is the number of words inthe sentence to be parsed, and m is the length of the rule. Since norule can not be of length more than a pre-defined constant, say 10,overall upper bound for this parsing procedure is linear—O(n).

For a triplet (Object, ObjectRole, RuleType) a set of verificationprocedures can be assigned. A procedure can be functional, e.g. “checkthat all non-auxiliary words in the word interval start with capitalletters”, or check that the word interval belongs to a particular listof collocations. For each new mapping, all applicable procedures areexecuted and, if one check fails the mapping is rendered incorrect. Atthis moment parser backtracks and generates the next partial mapping. Ifall checks are passed the parser adds next element into mapping andverification process starts again. Full mappings are stored to besupplied for next levels of verification such as cross reference orsemantic analysis.

In one embodiment of the present invention as illustrated in FIG. 10, amethod is presented for object, relationships and attributesidentification by providing mechanisms to iteratively verify thevalidity of a candidate for a new object, relationship or attribute.This method of the present invention defines recursive mechanisms thatverify the objects, relationships or attributes extracted by oneknowledge agent by finding a match with the objects, relationships orattributes appeared in the results of the analysis of other knowledgeagents. A rigorous use of these methods can virtually eliminate falsepositives. The algorithms are illustrated in determination of employeeposition and company name in Business Information Network.

To determine the validity of a potential object iterative bootstrappingprocedure is used.

One embodiment of iterative bootstrapping that can be utilized with thepresent invention is discussed hereafter. The same mechanism can be usedin different areas of object, relationship or attribute extractionwithin or outside Information Retrieval.

By way of illustration, and without limitation, consider the mentionedabove local grammatical rule: (“said”, PersonName, Employee, “,”,PositionName, “or”, CompanyName, Employer, “.”). If it is applied to aparticular sentence and the result of parsing is such thatPositionName=“Vice President of Operations” is already in the BusinessInformation Network, then the CompanyName of a particular parse isconsidered as a candidate for being included in Business InformationNetwork. But to be considered for the verification step, thisCompanyName should appear in a parse from a rule of different type that,say, puts different restrictions on the sequence of words to be aCompanyName. This process can be repeated several times to increase theassurance that this particular CompanyName is a valid one. And of courseif this particular CompanyName appeared in many more parses of differentdocuments, it increases the probability of it being valid. And, asusual, the set of dictionaries can be used to further verify thevalidity. The problem with dictionaries is that one needs to find theway to automatically build them starting with the core built which canbe manually. Dictionary of PositionNames is a good starting point due tothe relatively small size of it—thousands of entries vs., say, millionsof entries in the Dictionary of CompanyName's. The mentioned abovemechanism provides for that process. As soon as the dictionaries arelarge enough it is used quite aggressively to verify parses.

Business Information Network dictionaries include the dictionary ofPosition Names, Companies Names, Names of Individuals, and thedictionary of Synonyms, e.g. IBM—International Business Machine,Dick-Richard etc. These dictionaries grow along with the growth ofBusiness Information Network. Of course, people names and theirsynonyms/short versions are known pretty much in advance, as well as theofficial names of large companies, and a basic list of positions (e.g.President, CEO, Vice President of Marketing etc.). The bootstrappingprocess described above allow these dictionaries to grow based uponsuccessful parses with strict rules on potential validity of aparticular sequence of words to be a position or a company name; alsomanual verification is used when a low confidence value comes from thevalidity rules checker. This procedure does not guarantee 100%correctness of the dictionary entries, but is comes quite close to that.The random manual checks should be performed to lower false positiverate.

In one embodiment of the present invention as illustrated in FIG. 11, amethod is presented for extraction of PPCQ—Person, Position, Company,Quote—facts from individual news articles, press releases etc. A classicexample of PPCQ is—John Smith, VP of Marketing at XYZ said “ . . . ”.The list of potential companies being mentioned in the article caneither be furnished explicitly or can be implicitly presumed as beingfrom a known list of companies.

The PPCQ extraction algorithm can use the local grammar mechanismsdescribed elsewhere in this specification. These mechanisms extract thelist of candidate PPCQ vectors V=(person name, position, entity name,quote), which constitutes the initial set S of the PPCQ extractionalgorithm.

Often there is no one sentence that contains the full PPCQ. One sentencecan have PPC but no Q, another one has just person's first name andquote (John said “ . . . ”) or even a pronoun and a quote (she added “ .. . ”).

After the set S is built, the vectors related to the “same” person andthe “same” entity are merged, while “orphan” incomplete vectors andvectors with unclear attribution are excluded. This process is basicallya mapping between instances of the person-object and entity-object andthe corresponding objects. The names PINS and CINS are used for personand company instances (mentions) and PDEN and CDEN for correspondingobjects.

As illustrated in FIG. 11, the PPCQ extraction algorithm consists of thefollowing steps. Firstly, using the “C” part of PPCQ vectors from S theCINS set is built. Then by matching CINSs to the predefined explicit orimplicit list of companies the CDEN set is built. If a CINS belongs toseveral CDENs, it is excluded from further considerations. Then usingthe first “P” part of PPCQ vectors from S the PINS set is built.Similarly to CDEN, the PDEN set is built and PINSs that belong to morethan one PDEN are excluded. Then incomplete PPCQ vectors are merged tocreate four full components PPCQ vectors using direct component matchand pronoun resolution. Then for each PDEN maximum by inclusion positionis chosen. And finally all incomplete PPCQ vectors that were notembedded into full vectors are eliminated.

In one embodiment of the present invention as illustrated in FIG. 12, amethod is presented for extraction of VINs (Very Important Numbers) andassociated objects in unstructured and semi-structured documents.

The process of VINs extraction consists of the following steps. Firstly,the areas in the documents, where the numbers are mentioned aredetermined. Then these numbers are extracted, and finally the objectsthat these numbers are referring to are determined.

The areas containing VINs are defined by using layout format as well asthe grammatical structure. The layout information is used to detect thepotential VINs inside of a table or as a potential attribute of a page(e.g. copyright sign with dates at the bottom of a page) while asentence and paragraph syntactic structure is used in other cases.

VINs are described in several formats. A common one is the sequence ofdigits sometimes divided by comma. Also numbers can be spelled (liketwenty four instead of 24). The scale (%,$ etc.) is determined by theanalysis of the immediate surrounding.

To determine which object a particular VIN is referred to the followingmethods. If the VIN containing area is sentence, NLP parsing is appliedto determine the noun phrase corresponding to VIN. If the VIN containingarea has a structured format, such as list or table, the title of thelist or the corresponding column/row is used to determine the object.Thus for Business Information Network in the case of SEC filings grossrevenues are extracted from the tables, the row title is used todetermine a particular line item in the financials while the columntitle is used to determine the time interval such as quarter or year. Atthe same to determine the number of employees from the SEC filings theNLP parsing is used.

In one embodiment of the present invention as illustrated in FIG. 13,the bootstrapping process for building grammar rules for a particularvertical domain (Business Information Network, Travel, etc.) starts witha manual set of rules built by a knowledge engineer by observingdifferent types of documents and different ways of facts presentation.This zero iteration of rules is used by a fact extraction system togenerate a set of candidate facts, as described with regard islandgrammar herein.

The entities extracted from the zero iteration can be used to generatefirst level of iteration for a set of grammar rules using the followingprocess. The set separators used in each existing rule is enlarged byadding all “similar” words/collocations. Thus, if the pronoun “he” is inthe set, than the pronoun “she” is added to the same separator. The sameprocess is applied to different tenses of the verbs (e.g. the verb“said” generates “says” and “say”), particles (e.g. “on” generates“off”, “in” etc.). And finally all synonyms of the existing separatorsare added too.

Then the expanded grammar is applied to a large number of representativepages (e.g. if one press release was parsed by the existing grammar, addall press releases from the same company, or from the entire servicelike Business Wire) to extract facts. Separators that did notparticipate in the extracted facts are deleted from the grammar, unlessthey were present at the zero iteration. They also are deleted if theyproduced a lot of erroneous results.

Then a set of new pages is presented for the fact extraction using thenew version of separators. New objects and attributes, e.g. Position,CompanyName, PersonName, which participated in the extracted facts areadded to the object dictionaries.

The second, third, and so on iterations can be done in the same way. Thenumber of iterations depends on the quality of the initial set of rulesand the size of the training set of documents. The process can stop, forexample, after 10 or so iterations due to stabilization of the grammaror when it reaches a pre-defined maximum number of iterations.

Due to the high efficiency (O(n)) of the parsing mechanism evenwords/separators from the set of rules that were used rarely or evenwere not used at all in the training set are kept in the grammar. Thisapproach makes the set of rules quite stable and minimizes themaintenance problem. It also helps to deal with unseen before factdescription habits.

In one embodiment of the present invention as illustrated in FIG. 14, amethod is presented for object identification and inference. Theapproach is based on three-layer representation of an object (Instance,Denotatum, Denotatum Class), the roll forward mechanism to deleteincorrect equivalences without destroying correct ones. Also the methodsof inference based on morphological, grammatical and layout proximitybetween instances of the objects and their unique attributes arepresented.

By way of illustration, and without limitation, Business InformationNetwork deals with the Instance-Denotatum problem for each object,company and person. In this embodiment, Business Information Network hasthree levels of representation. The first level is an “instance” level.Corresponding types are CINS and PINS for instances of companies andpersons. Each sequence of words in a document that can be a name of anobject, e.g. CINS, is stored as an instance of an object, which iscalled its denotatum, CDEN. Each document is presumed not havingequivalent CINS's belonging to different CDEN's. For example, one cannot use in one document the same name for two different companieswithout creating confusion. The problem becomes more complicated whenone goes beyond individual document. The equivalence of two differentCDEN's can be determined using different heuristics similar to the onejust described. But the very nature of dynamic Business InformationNetwork facts extraction process demands that the equivalence can bedetermined and reevaluated. That's why Business Information Networkcontains the third level, so-called DENClass that provide necessarymeans for denotata equivalence.

In one embodiment, the inference rules are divided into domain dependentand domain-independent. An example of domain-dependent rule is the rulethat a person can be a member of several Boards of Directors but can notbe a vice president of two different companies at the same time. Thisrule is not absolute, so if there are many facts about a particularperson that say otherwise this rule can be suppressed. The suppressionusually happens if there is no temporal information available, since inmost cases these positions were held not simultaneously. Using the timestamp extraction mechanism of the present invention can resolve issueslike this in many cases. Time stamps also help in building person's biofrom disparate facts collected from different sources. Another way toresolve the potential contradiction is determination of verb tense beingused to describe the fact. Thus, in press releases often the phraseslike “before joining our company, John Smith was a director of marketingcommunications at Cisco” are quite useful not only to build a bio, butalso to distinguish this John Smith from another one with the same namewho did not work at Cisco before joining this company. Absolute orrelative temporal information like this constitutes a domain-independentinference rule.

New facts can be added to the fact database constantly. These facts canbring new information, can be a change of the existing facts includingtheir invalidation, e.g. retraction of a publication, and can also be incontradiction with the existing facts. Besides the facts are coming inno particular order due to parallel search and multiple sources thatgenerate the facts. To deal with the problem of potential errors andcontradictions in entities extraction and equivalence determination, oneembodiment of the present invention as illustrated in FIG. 15, applies anon-traditional transactional model called “Roll Forward”. If acontradiction or error in equivalence is determined, which can happendue to a human reporting an error or due to contradictive factscollected automatically, the “suspicious” area is “disassembled” and“reassembled” again. A typical example is incorrect “merging” of twopersons with the same name into one person. If that error is detectedthe entire PDENClass is destroyed, and two new PDENClasses are builtfrom scratch using all PDENs that belonged to the destroyed PDENClass.This mechanism is especially effective when the concept of candidatefacts is propagated through the architecture of the system. The decisionif two instances of the “John Smith” represent the same person or not ismade in the following two ways. One way is to use a system default indetermining the correspondence between INS, DEN and DENClass, whileanother one is to provide a user with the parameters to determine thescope of sources and the threshold of the “merging” decision. Thus, ifthe user has a preference of the sources that contain “correct” facts,the number of potential “merging” errors can be reduced significantly.Also parameters like time stamp, position, company name, school name,can be used to make the “merging” decision.

In one embodiment of the present invention as illustrated in FIG. 16, aBusiness Information Network is defined as a hyper graph consisting oftwo types of major objects, companies and individuals. Each object hasits own list of attributes and objects are connected with each other bydifferent relationships, e.g. employee-employer, company-subsidiary,vendor-customer, seller-buyer, etc.

The system 10 of the present invention can provide a new way to look atthe economy in general as well as at a particular industry or marketsegment. Knowing the relationships between companies one can obtainanswers to questions about market segment activity, trends, acceptanceof new technology, and so on and so forth. The system 10 can be utilizedin a variety of different uses, including but not limited to, provideon-line service to sales people to help them better assess prospects andfind right people in prospects to approach for potential sale, use asventure capital investment strategy based on the knowledge of the smallcompanies' activities and buying patterns of large companies, in mergerand acquisition activity where the system 10 facilitates the process offinding a buyer for a company or a target for acquisition, and the like.

In one embodiment as illustrated in FIG. 17, the process of generating aBusiness Information Network database can consist of the followingsteps. Firstly, the documents from different sources are collected. Thesources include Public Internet Companies/Organizations web sites, PressReleases, Magazines/Journals Publications, Conferences Presentations,Professional Memberships Publications, Alumni News, Blogs etc.;Government Sources—SEC Filings, USPTO, Companies Registration, etc.;Proprietary Sources (to be used only by the users that provided them orauthorized to by the owner)—Magazines/Journal Publications, PurchasedDatabases, Analyst Reports, Purchased Trade Shows Attendance Lists,etc.; Personal Rolodexes (to be used only by a person who provided it);Companies' intranets and databases (to be used only by the peopleauthorized by the information owner). Then knowledge agents are appliedto documents to extract business related information to be stored inBusiness Information Network Database. After that incorrect orirrelevant facts are filtered out using different fact verificationtechniques. Then different consistency checks are applied to solidifythe correctness of facts. The facts that went through these checks arestored in Business Information Network database. Then the information inthe database is made available to on-line users. The collection processconstitutes permanent activity, since the information grows every day,and changes every day.

In one embodiment of the present invention, a business informationsystem is provided that extracts facts deals with the issue of efficientpresentation of these facts in a structured form. The objects, theirrelationships and their attributes should be stored in a way to make theprocess of answering questions straightforward and efficient. To be ableto do that the data representation should reflect potential questions.At the same time the data representation should be relevant to themechanisms for facts extraction, since they Ultimately decide whatinformation is stored in the repository. In one embodiment of thepresent invention, a method is presented for designing of templates thatcovers majority of business questions, and building database structurethat supports these templates and at the same time matches thecapability of the facts extraction mechanisms described in relatedsections. Business Information Network frameworks can include thefollowing elements: objects companies, individuals; relations:subsidiary, acquisition, employee, employer, friend, vendor partner,customer, schoolmate, colleague; auxiliary elements: —paragraphs,documents, web pages; attributes—position, quote, earnings, address,phone number; instances and denotate.

There are two major objects in Business Information Network—company andindividual. Company object represents businesses, non-profitorganization, government entities and any other entities thatparticipate in one way or another in economic activity.Individual/Person object represents any person participating in economicactivity, such as employee, owner, government official etc.

Objects can participate in relationships. Each relationship has twoobjects that are a part of it. Different relationships extracted fromthe same document are useful to establish multi-link relations. Forexample, a quote in a press release can establish that a person works ata company that is a vendor of another company. Auxiliary elementsinclude web pages, documents (can be several in one page) and paragraphs(can be several in one document).

Each object, relationship or auxiliary element can have attributes.Attributes can be static, e.g. time stamp, URL, and dynamic, e.g.position, quote.

As an illustration consider the following example. A press release thatcontains the following information: “Company C purchased a Product Pfrom Company V. The Product P is installed in X number of locations.Person V, VP Sales of Company V is “delighted to have Company C as acustomer of their new line of products” and Person C, CIO of Company Cis “considering Product P the first step in their 3 year project torevamp the entire IT infrastructure of Company C” will yield thefollowing relationships:

Relationship Object Type Types Objects Attributes Employer - CINS-Company C - Person C Position: CIO Employee PINS Quote: “” Employer -CINS- Company V - Person V Position: VP Sales Employee PINS Quote: “”Customer - CINS- Company C - Company V Product: P Vendor CINS VIN: Xnumber of locations Quotes Customer - CINS- Company C - Person V QuoteSeller PINS Vendor - CINS- Company V - Person C Quote Purchaser PINS

In one embodiment of the present invention the list of attributesincludes the following: company—name, address/phone/url, about,quarterly/early sales, number of employees; offering—name, description;person—name, age; relationships employee-employer—position, time stamp;vendor-customer—quote, time stamp; company-acquirer—quote, time stamp;member-association—quote, time stamp.

In one embodiment of the present invention as illustrated in FIG. 18, aconcept of Implicit Social Network is introduced and a method ispresented for building it by analyzing unstructured documents, and/ordirectly using Business Information Network.

To address the problems of explicit rolodex described above, oneembodiment of the present invention is an Implicit Social Network. Twopeople are connected implicitly if they have some of the followingthings in common: they worked on the same board for some time interval;they were members of the same management team for some time interval;they graduated the same year from the same graduate school; they werebuyers and sellers, correspondingly, in the same transaction. There aremany other cases when two people know each other, but not necessarilykeep the name of another person in their corresponding rolodexes.

Each particular type of relations can be more or less strong and more orless relevant to a task of a person trying using Implicit SocialNetwork. The Implicit Social Network exists side by side with ExplicitRolodex and quite often overlaps it. The advantages of Implicit SocialNetwork come from the fact that it is built using publicsources—Internet first and foremost. As a result, it is completelytransparent, it potentially can include tens of millions of people, itupdates on a daily basis.

Implicit Social Network is represented as a graph of individuals withedges colored by the type of connection and weighed by the number offactors defining the type of connection. For example, with the work onthe same management team the duration is an important factor. Also iftwo people worked together as members of management team in severaldifferent companies the weight of the edge is much higher than if theyworked together for few months just once.

Implicit Social Network is a subgraph of Business Information Networkgraph that consists of individual-individual relationships withattributes defining the details of the relationships between twoindividuals and weight function defining the strength of therelationship. The strength and importance of the relationshipincorporates objective (e.g. time spent working together) anduser-defined parameters (e.g. only work in telecommunication industry isrelevant).

The world of business relations can be described as a temporal coloredgraph G with two types of vertices—people and companies. The colors ofedges between people vertices represent social networking relationships.The colors of edges between companies represent relationships likepartners, vendors, customers, etc. The colors of edges between peopleand companies represent relationships like employee, consultant,customer, etc. Temporal portion of this graph is represented by a pairof time stamps (from, to) associated with each vertex and each edge. Anumber of questions about business can be expressed in terms of thisgraph and answered by a system (like Business Information Network) thathas this graph populated. These questions are covered by Customer AlumniNetwork.

As illustrated in FIG. 19, a Customer Alumni Network for a particularcompany, called nucleus, is a set of people that worked for thiscompany's customers in specified position in a specified time intervalplus the companies they work for now. Without using this particularterm, sales people were looking for capitalizing on their marqueeaccounts to acquire new customers using people that had first handexperience with their product and can be champions if not decisionmakers in their new jobs. Customer Alumni Network is built directly fromBusiness Information Network starting with nucleus and going through itscustomers, then buyers and employees in these customers and into theirnew employments after they left these customers of nucleus.

While embodiments of the invention have been illustrated and described,it is not intended that these embodiments illustrate and describe allpossible forms of the invention. Rather, the words used in thespecification are words of description rather than limitation, and it isunderstood that various changes may be made without departing from thespirit and scope of the invention.

What is claimed is:
 1. A method for reducing a number of false positivesin a fact extraction process, comprising: utilizing a deferred decisionmethod; building several levels of verification steps to eliminateincorrect candidate facts; applying a set of knowledge agents ofdifferent type to verify results of each other; utilizing N-best resultsto support a deferred decision method; and accommodating a detailedrerun of appropriate knowledge agents to retrieve additional parametersto resolve fact discrepancies.
 2. A method for extraction of veryimportant numbers and objects that are associated with from a pagecomprising: determining paragraphs and tables containing numbers; andextracting the numbers.
 3. The method of claim 2, wherein immediatecontext of each extracted number comprising of a paragraph or a table isdetermined.
 4. The method of claim 3, wherein classic part of speechtagging is applied to determine the object corresponding to the number,if the context is a paragraph.
 5. The method of claim 4, wherein thetable column titles are used to determine the column describing theobject corresponding to the column containing the number, if the contextis a table.
 6. The method of claim 5, wherein the value of the cell inthe determined object column and in the number's row is chosen as theobject corresponding to the number.
 7. The method of claim 5, whereinrows are used instead of columns to determine the object position.